{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DenseNet_implementation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyM1SJiQass0yTKMfF93h63w",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Machine-Learning-Tokyo/CNN-Architectures/blob/master/Implementations/DenseNet/DenseNet_implementation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IIC87opM61Qm",
        "colab_type": "text"
      },
      "source": [
        "# Implementation of DenseNet\n",
        "\n",
        "We will use the [tensorflow.keras Functional API](https://www.tensorflow.org/guide/keras/functional) to build DenseNet from the original paper: “[Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993)” by Gao Huang, Zhuang Liu, Laurens van der Maaten, Kilian Q. Weinberger.\n",
        "\n",
        "[Video tutorial](https://www.youtube.com/watch?v=3ZPJyknZolE&list=PLaPdEEY26UXyE3UchW0C742xh542yh0yI&index=8)\n",
        "\n",
        "---\n",
        "\n",
        "In the paper we can read:\n",
        "\n",
        ">**[i]** “Note that each “conv” layer shown in the table corresponds the sequence BN-ReLU-Conv.\"\n",
        ">\n",
        ">**[ii]** \"[...] we combine features by concatenating them. Hence, the $\\ell th$ layer has $\\ell$ inputs, consisting of the feature-maps of all preceding convolutional blocks.\"\n",
        ">\n",
        ">**[iii]** \"If each function $H_\\ell$ produces $k$ feature-maps, it follows that the $\\ell th$ layer has $k_0 + k × (\\ell − 1)$ input feature-maps, where $k_0$ is the number of channels in the input layer.\"\n",
        ">\n",
        ">**[iv]** \"The initial convolution layer comprises 2k convolutions of size 7×7 with stride 2\"\n",
        ">\n",
        ">**[v]** \"In our experiments, we let each 1×1 convolution produce 4k feature-maps.\"\n",
        ">\n",
        ">**[vi]** \"If a dense block contains m feature-maps, we let the following transition layer generate $\\lfloor \\theta m \\rfloor$ output feature-maps, where $0< \\theta ≤1$ is referred to as the compression factor. [...] we set $\\theta$ = 0.5 in our experiment.\"\n",
        "\n",
        "---\n",
        "\n",
        "We will also make use of the following Table **[vii]** and Diagram **[viii]**:\n",
        "\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/DenseNet/DenseNet.png width=\"90%\">\n",
        "<img src=https://raw.githubusercontent.com/Machine-Learning-Tokyo/DL-workshop-series/master/Part%20I%20-%20Convolution%20Operations/images/DenseNet/DenseNet_block.png width=\"50%\">\n",
        "\n",
        "---\n",
        "\n",
        "## Network architecture\n",
        "\n",
        "We will implement the Dense-121 (k=32) version of the model (marked with red in **[vii]**).\n",
        "\n",
        "The model:\n",
        "- starts with a Convolution-Pooling block\n",
        "- continues with a series of:\n",
        " -- Dense block\n",
        " -- Transition layer\n",
        "- closes with a *Global Average pool* and a *Fully-connected* block.\n",
        "\n",
        "<br>\n",
        "\n",
        "In every Dense block the input tensor passes through a series of *conv* operations with fixed number of filters (*k*) and the result of each one is then concatenated to the original tensor **[ii]**. Thus the number of feature maps of the input tensor follows an arithmetic growth at every internal stage of the Dense block by *k* tensors per stage **[iii]**.\n",
        "\n",
        "In order for the size of the tensor to remain manageable the model makes use of the ***Transition layers***.\n",
        "\n",
        "At each *Transision layer* the number of feature maps of the input tensor is reduced to half (multiplied by $\\theta=0.5$) (**[vi]**).\n",
        "\n",
        "Also the spatial dimensions of the input tensor are halved by an *Average Pool* layer (**[vii]**).\n",
        "\n",
        "### Dense block\n",
        "At each Dense block we have a repetition of:\n",
        "- 1x1 conv with $4\\cdot k$ filters\n",
        "- 3x3 conv with k filters\n",
        "\n",
        "blocks.\n",
        "\n",
        "As it is written in **[i]**: \n",
        ">each “conv” layer corresponds the sequence BN-ReLU-Conv\n",
        "\n",
        "---\n",
        "\n",
        "## Workflow\n",
        "We will:\n",
        "1. import the neccesary layers\n",
        "2. write the *BN-ReLU-Conv* function (**[i]**)\n",
        "3. write the *dense_block()* function\n",
        "4. write the *transition_layer()* function\n",
        "5. use the functions to build the model \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ajn4ZhcL7OcS",
        "colab_type": "text"
      },
      "source": [
        "### 1. Imports"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I-Friz_-7Nts",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow\n",
        "from tensorflow.keras.layers import Input, BatchNormalization, ReLU, \\\n",
        "     Conv2D, Dense, MaxPool2D, AvgPool2D, GlobalAvgPool2D, Concatenate"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S6b4YIHQ7V8y",
        "colab_type": "text"
      },
      "source": [
        "### 2. BN-ReLU-Conv function\n",
        "The *BN-ReLU-Conv* function will:\n",
        "- take as inputs:\n",
        "    - a tensor (**`x`**)\n",
        "    - the number of filters for the *Convolution layer* (**`filters`**)\n",
        "    - the kernel size of the *Convolution layer* (**`kernel_size`**)\n",
        "- run:\n",
        "    - apply *Batch Normalization* to `x`\n",
        "    - apply ReLU to this tensor\n",
        "    - apply a *Convolution* operation to this tensor\n",
        "- return the final tensor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qm39e--77Yoz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def bn_rl_conv(x, filters, kernel_size):\n",
        "    x = BatchNormalization()(x)\n",
        "    x = ReLU()(x)\n",
        "    x = Conv2D(filters=filters,\n",
        "               kernel_size=kernel_size,\n",
        "               padding='same')(x)\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ud_TpqOH7dc3",
        "colab_type": "text"
      },
      "source": [
        "### 3. Dense block\n",
        "\n",
        "We can use this function to write the *Dense block* function.\n",
        "\n",
        "This function will:\n",
        "- take as inputs:\n",
        "    - a tensor (**`tensor`**)\n",
        "    - the filters of the conv operations (**`k`**)\n",
        "    - how many times the conv operations will be applied (**`reps`**)\n",
        "- run **`reps`** times:\n",
        "  - apply the 1x1 conv operation with $4\\cdot k$ filters (**[v]**)\n",
        "  - apply the 3x3 conv operation with $k$ filters (**[iii]**)\n",
        "  - *Concatenate* this tensor with the input **`tensor`**\n",
        "- return as output the final tensor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fhWKuM_n7dDv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def dense_block(tensor, k, reps):\n",
        "    for _ in range(reps):\n",
        "        x = bn_rl_conv(tensor, filters=4*k, kernel_size=1)\n",
        "        x = bn_rl_conv(x, filters=k, kernel_size=3)\n",
        "        tensor = Concatenate()([tensor, x])\n",
        "    return tensor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mp3M2Bqx7jF_",
        "colab_type": "text"
      },
      "source": [
        "### 4. Transition layer\n",
        "Following, we will write a function for the transition layer.\n",
        "\n",
        "This function will:\n",
        "- take as input:\n",
        "  - a tensor (**`x`**)\n",
        "  - the compression factor (**`theta`**)\n",
        "- run:\n",
        "  - apply the 1x1 conv operation with **`theta`** times the existing number of filters (**[vi]**)\n",
        "  - apply Average Pool layer with pool size 2 and stride 2 (**[vii]**)\n",
        "- return as output the final tensor\n",
        "\n",
        "Since the number of filters of the input tensor is not known a priori (without computations or hard coded numbers) we can get this number using the `tensorflow.keras.backend.int_shape()` function.\n",
        "This function returns the shape of a tensor as a tuple of integers\n",
        "\n",
        "In our case we are interested in the number of feature maps/filters, thus the last number [-1] (channel last mode)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vcyLpZu37izC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def transition_layer(x, theta):\n",
        "    f = int(tensorflow.keras.backend.int_shape(x)[-1] * theta)\n",
        "    x = bn_rl_conv(x, filters=f, kernel_size=1)\n",
        "    x = AvgPool2D(pool_size=2, strides=2, padding='same')(x)\n",
        "    return x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sJciWUyu7oMq",
        "colab_type": "text"
      },
      "source": [
        "### 5. Model code\n",
        "Now that we have defined our helper functions, we can write the code of the model.\n",
        "\n",
        "The model starts with:\n",
        "- a Convolution layer with $2\\cdot k$ filters, 7x7 kernel size and stride 2 (**[iv]**)\n",
        "- a 3x3 Max Pool layer with stride 2 (**[vii]**)\n",
        "\n",
        "and closes with:\n",
        "- a Global Average pool layer\n",
        "- a Dense layer with 1000 units and *softmax* activation (**[vii]**)\n",
        "\n",
        "Notice that after the last *Dense block* there is no *Transition layer*.\n",
        "For this we use a different letters (d, x) in the `for` loop so that in the end we can take the output of the last *Dense block*."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O7y86ANc7n4A",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "k = 32\n",
        "theta = 0.5\n",
        "repetitions = 6, 12, 24, 16\n",
        " \n",
        "input = Input(shape=(224, 224, 3))\n",
        " \n",
        "x = Conv2D(2*k, 7, strides=2, padding='same')(input)\n",
        "x = MaxPool2D(3, strides=2, padding='same')(x)\n",
        " \n",
        "for reps in repetitions:\n",
        "    d = dense_block(x, k, reps)\n",
        "    x = transition_layer(d, theta)\n",
        " \n",
        "x = GlobalAvgPool2D()(d)\n",
        " \n",
        "output = Dense(1000, activation='softmax')(x)\n",
        " \n",
        "from tensorflow.keras import Model \n",
        "model = Model(input, output)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V1Kb_F4_8BwP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.utils import plot_model\n",
        "plot_model(model, show_shapes=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F3i54nPv7uJQ",
        "colab_type": "text"
      },
      "source": [
        "## Final code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5-F-vsU67t2Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow\n",
        "from tensorflow.keras.layers import Input, BatchNormalization, ReLU, \\\n",
        "     Conv2D, Dense, MaxPool2D, AvgPool2D, GlobalAvgPool2D, Concatenate\n",
        " \n",
        " \n",
        "def bn_rl_conv(x, filters, kernel_size):\n",
        "    x = BatchNormalization()(x)\n",
        "    x = ReLU()(x)\n",
        "    x = Conv2D(filters=filters,\n",
        "               kernel_size=kernel_size,\n",
        "               padding='same')(x)\n",
        "    return x\n",
        " \n",
        " \n",
        "def dense_block(tensor, k, reps):\n",
        "    for _ in range(reps):\n",
        "        x = bn_rl_conv(tensor, filters=4*k, kernel_size=1)\n",
        "        x = bn_rl_conv(x, filters=k, kernel_size=3)\n",
        "        tensor = Concatenate()([tensor, x])\n",
        "    return tensor\n",
        " \n",
        " \n",
        "def transition_layer(x, theta):\n",
        "    f = int(tensorflow.keras.backend.int_shape(x)[-1] * theta)\n",
        "    x = bn_rl_conv(x, filters=f, kernel_size=1)\n",
        "    x = AvgPool2D(pool_size=2, strides=2, padding='same')(x)\n",
        "    return x\n",
        " \n",
        "\n",
        "k = 32\n",
        "theta = 0.5\n",
        "repetitions = 6, 12, 24, 16\n",
        " \n",
        "input = Input(shape=(224, 224, 3))\n",
        " \n",
        "x = Conv2D(2*k, 7, strides=2, padding='same')(input)\n",
        "x = MaxPool2D(3, strides=2, padding='same')(x)\n",
        " \n",
        "for reps in repetitions:\n",
        "    d = dense_block(x, k, reps)\n",
        "    x = transition_layer(d, theta)\n",
        " \n",
        "x = GlobalAvgPool2D()(d)\n",
        " \n",
        "output = Dense(1000, activation='softmax')(x)\n",
        " \n",
        "from tensorflow.keras import Model \n",
        "model = Model(input, output)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nc_LI1Od7Jjw",
        "colab_type": "text"
      },
      "source": [
        "## Model diagram\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/Machine-Learning-Tokyo/CNN-Architectures/master/Implementations/DenseNet/DenseNet_diagram.svg?sanitize=true\">"
      ]
    }
  ]
}